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Abstract. We investigate security properties of the Anshel-Anshel-Goldfeld 
commutator key-establishment protocol [I] used with certain polycyclic groups 
described in [3]. We show that despite low success of the length based attack 
shown in [5] the protocol can be broken by a deterministic polynomial-time 
algorithm. 
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1. Introduction 


In this paper we analyze the commutator key-establishment protocol [I] used 
with certain polycyclic groups described in [ 3 ]. The commutator key-establishment 
(CKE) protocol is a two-party protocol performed as follows. 

• Fix a group G (called the platform group) and a set of generators 
gi,... ,gk for G. All this information is made public. 

• Alice prepares a tuple of elements a = (ai,..., a^i) called Alice’s public 
tuple. Each ai is generated randomly as a product of gt^s and their inverses. 

• Bob prepares a tuple of elements b = {bi,..., 67V2) called the Bob’s public 
tuple. Each bi is generated randomly as a product of ^i’s and their inverses. 

• Alice generates a random element A as a product a®) ... of at ’s and 
their inverses. The element A (or more precisely its factorization) is called 
the Alice’s private element. 

• Bob generates a random element B as a product of 6i’s and their 

inverses, called the Bob’s private element. 

—A 

• Alice publishes the tuple of conjugates b = {A~^biA ,..., 

• Bob publishes the tuple of conjugates = (i?“^aii?,..., 

• Finally, Alice computes the element as a product: 


A-^ ■{B-^allB...B-^al^^B) 

using the elements of Bob’s conjugate tuple . 
• Bob computes the key Kb as a product: 



using the elements of Alice’s conjugate tuple b 


Date: April 21, 2015. 

The second author has been partially supported by NSA Mathematical Sciences Program grant 
number H98230-14-1-0128. 


1 



2 


M. KOTOV AND A. USHAKOV 


It is easy to check that Ka = Kb = in G. The obtained commutator 

is the shared key. 

Security of the commutator key establishment protocol is based on computational 
hardness of computing the commutator [^4, B] based on the intercepted public infor¬ 
mation - the tuples a, b and their conjugates ,b .In practice it is often achieved 
by solving systems of conjugacy equations for A and B, i.e., finding X = A! and 
Y = B' satisfying: 

X-^hiX = b[, 

and 

X-^bN,X = b'j^^, 

and computing K' = [A' ^ B']. In general it can happen that K' ^ K as explained 
in m, but as practice shows very often K = K' (for instance, as in [B]). 

A big advantage of the commutator key-establishment protocol over other group- 
based protocols is that it can be used with any group G satisfying certain compu¬ 
tational properties. Originally, the group of braids was suggested to use as a 
platform group, but after a series of attacks it became clear that B^ can not pro¬ 
vide good security. But the search for a good group is still very active and in [5] a 
certain class of polycyclic groups was proposed to be used with CKE. In this paper 
we show that that class can not provide good security. For more on group-based 
cryptography see [9] . 

l.I. Outline. In Section [5] we define the class of groups under investigation and 
discuss two different ways to represent the elements. In Sections[3]and[4]we describe 
the attacks on different group presentations. 

2. The platform group 

Consider an irreducible monic polynomial f{x) G Z[x] and define a field: 

F = Q[x]/{f). 

The ring of integers of F is dehned as: 

Op = {a G F I a is a zero of a monic polynomial g{x) G Z[x]} 

and its group of units: 

Up = {a I a ^ G Gp}. 

A semidirect product Up k Op of Up and Op is defined as a Cartesian product 
Up X Op equipped with the following binary operation: 

(1) (a,a) • (/3,6) = (a/3,a/3-b6). 

The constructed group Gp is the platform group in [3]. It is easy to see that Gp is 
polycyclic and metabelian and there are several different ways to represent Gp. 

(a) One can work with Gp as it is defined above, i.e., as a semidirect product, 
in which case its elements are represented as pairs and multiplication o is 
used. 

(b) One can construct a polycyclic presentation for Gp and work with its ele¬ 
ments as with words over the generating set. 

Unfortunately, neither [3] nor give any detail on how to treat Gp. Since com¬ 
putational properties of the same group can vary depending on a way we represent 
its elements, in the next sections we discuss both presentations of Gp. 


Y-'^aiY = a'l, 

= a(v^. 
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2.1. Gf as a set of pairs of matrices. There are different ways to represent 
the elements of F. For instance, elements in F can be represented as polynomials 
over Q of degree up to n — 1 with addition and multiplication performed modulo 
the original polynomial /. Also one can represent elements in F by matrices as 
described below. Recall that the companion matrix for a monic polynomial 
/ = x" + Cn-ix^~^ + ... + cix + Co is a matrix of the form: 


M = 


0 0 
1 0 
0 1 


0 -Co 

0 —Cl 
0 —C2 


0 0 ... 1 -c„_i 


The characteristic and minimal polynomial of M is / and the set of matrices: 

( 2 ) F = ^QqE + aiM + 02^/^ + . . . + — ^ \ Qq, ..., fln—i S Q} . 

equipped with the usual matrix addition and multiplication is the field F. The 
correspondence between two presentations is obvious: 

CLq T CLiX T . . . T CL^i — lX^ ^ i ^ Uq T (l\M T . . . -t- 


and choosing a particular presentation we do not change computational properties 
of F. Here we choose matrix presentation for F. 

Let Oi,, On be a basis of the ring of integers Op: where each Oi is a matrix. 
Hence: 

Of = {o-lOi + 0202 + . . . + CLnOn | Oi, . . . , O^ € Z} . 

Let {Hi,..., Um} be a generating set for the group Up, where every Hi is a matrix. 
Hence: 

Up = {H“^ • H“= ■... • H“™ I oi,... ,a™ € Z} . 

By Dirichlet theorem [71 Chapter 8] Up = Zj, x Z'"”^, where m = s + t— 1, s is 
the number of real field monomorphisms F ^ R, and 2t is the number of complex 
field monomorphisms F ^ C. Without loss of generality it can be assumed that 
Hf = E. 

Now naturally the group Gp = Up x Op is a set of pairs of matrices: 
G={{G,S)\G eUp.S eOp}, 
equipped with multiplication given by: 

(3) {C,S)-iD,T) = {GD,SD + T). 

It is easy to check that the inverse in Up x Op can be computed as 

(4) (0,5)-i = (0-1,-50-1), 

which gives the following expression for the conjugate of {B,T) by (0,5) 

(5) (H,T)(^’®) = (0,5)-i(H,T)(0,5) = (H,5(F - H) + TO), 


where E is the identity matrix. 
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2.2. Gf given by polycyclic presentation. Recall that a group G is called 
polycyclic if there exists a subnormal series of G: 

G = Go>G2>G3.-.>G„ = {1}, 


with cyclic factors Gi-xjGi. Denote [Gi_i : Gi] by and put I = {i \ n < oo}. 
Relative to the series above one can find a generating set gi,... ,gn for G satisfying 
(Gi^gi) = Gi-i- Every element g G G can be uniquely expressed as a product 
g = gl^ ... g®", where G Z, i = 1,.. ., n, and 0 < Cj < if i G /. The polycyclic 
group G has a hnite presentation of the form: 


( 6 ) 



gf = Wij,gf = Uy for 1 < i < j < n, 
gf = Uk for fc G / 


where Wij, Vij, and Ui are words in ..., This presentation is called a 
polycyclic presentation. For more details see [3 Chapter 8]. 

It is straightforward to find a polycyclic presentation for the group Gp = Up ^ 
Op. It has generators gi,..., ffm, 5m+i, • ■ ■, 3m+n, where gi,...,gm correspond to 
the pairs (C/i, O),..., (1/^, O) G UpKOp (O is the zero matrix), and gm+i^ ■ ■ ■, gm+n 
correspond to the pairs {E, Of,..., {E, 0„) G Up « Op {E is the identity matrix). 
The set of relations for G is formed as follows. 

• gf+j = gf+i ■ ■ ■ gf+ni i = l, • ■ •, m, j = l,..., n, and 1 ,..., a,jn are the 
coefficients in the expression OjUi = ttyiOi + ... + aijnOn, 

• gf+j = gtfh ■ ■ ■ g’ffn^ i = l,...,m,j = l,...,n,and ..., are the 

coefficients in the expression OjU~^ = fcyiOi + ... + bijnOn, 

• gi=e, 

• [gi^gA = e, l <i < j <m, 

• [gi, gj] = e, m + 1 < i < j < m + n. 


3. Attack on semidirect product 


In this section we assume that the group Gf is given as a semidirect product and 
the field E is described using matrices as in (I2|). The general idea behind the attack 
is to extend the group Gf and work in G*p = E* k F. The group G*p is, in general, 
not finitely generated and hence is not polycyclic. Nevertheless the elements of G* 
can be effectively represented by pairs of matrices as described in Section ITT] 
Consider a system of conjugacy equations related to the Alice’s private key: 

r A-ifciA = b[, 

(7) 

[ X-^bN,X =bf^, 

with unknown X G Up t< Op. We treat the system as a system over F* t< F and 
hence: 

A = (G,5), b, = b[ = in F* k F. 

Using ([5]) we get the following system of N 2 linear equations over the held F with 
two unknowns G and S: 


S{E-Bf + TiG = T[, 

— T' 

— ^N2- 


(8) 


S{E-Bn2) + Tn2G 
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It has a unique solution when the coefficient matrix of the system has rank 2 over 
the field F, in which case the obtained solution A' is the same as the original Alice’s 
private key. We call the described approach “field based attack” or simply FBA. 

The described attack was implemented in GAP [4]. Its implementation can be 
found in [8]. The table below compares success rate and time efficiency of our 
attack and the attack in [5]. Our tests were run on Intel Core i5 1.80GHz computer 
with 4GB of RAM, Ububtu 12.04, GAP 4.7. 


Polynomial 

KG) 

LB A w/ dynamic 
set, L = 5 

FBA, L = 5 

FBA, L = 100 

Time 

Success 

rate 

Time 

Success 

rate 

Time 

Success 

rate 

x‘^ — X — 1 

3 

0.20 h 

100% 

2.4 s 

100% 

2.8 s 

100% 

x^ - x^ - 1 

7 

76.87 h 

35% 

3.4 s 

100% 

5.3 s 

100% 

x"^ — x^ — 1 

10 

94.43 h 

8% 

5.2 s 

100% 

9.7 s 

100% 

a;® -7x^-1 

14 

95.18 h 

5% 

23.1 s 

100% 

57.7 s 

100% 

x^^ — x^ — 1 

16 

95.05 h 

5% 

15.3 s 

100% 

29.5 s 

100% 

x^^ — X — 2 

22 

- 

- 

694.8 s 

100% 

607.4 s 

100% 

— a: — 1 

30 

- 

- 

208.5 s 

100% 

192.8 s 

100% 


The first four columns of this table are taken from [5] . For our tests we used the 
same parameter values: Ni = N 2 = 20, and the same number of tests: 100. 


4. Attack on polycyclic presentation 

In this section we assume that Gp is given by a polycyclic presentation described 
in Section o First we show that the group Gp can be presented as a seinidirect 
product of an abelian matrix group and Z". Then we present the attack on the 
obtained presentation. 


4.1. Deduced semidirect product for Gp. Given a polycyclic presentation for 
Gp constructed in Section 12.21 it is straightforward to find the numbers m and n. 
For the relations: 

Qi _ O'ijn 

9m+j 9m+l ■ * ■ 9m+n 
we can define matrices Ci,..., Cm- 


Ci — {o,ijk)j=p 


.,n 

.,n ■ 


Next we form a semidirect product G of (Ci,..., Gm) and Z" which is a set of 
pairs: 

{(C,!) I Ce (Ci,...,C'™),sgZ"} 
equipped with the multiplication given by 


(C,s) • {D,t) = iCD,sD + t). 


Let {ei,..., e„} be the standard basis for Z”. It is easy to check the map t : {gi, ..., gm+n} 
G given by: 

^ f (^,0) if * < m, 

\iE,ej) a i = m +j, 1 < j <n, 

defines an isomorphism between Gp and the constructed group. Furthermore, given 
an element g = gl^ ... g^" it requires polynomial time to find its r-image. 
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We also claim that given a pair {C,v) it requires polynomial time to find a 
word g such that r(g) = {C,v). To convert {C,v) into a word in the generators 
gi,, gm+n one can express (C,v) as a product: 

(C,u) = (Ci, 0 )“i ... 

for some oi,..., Qn+m G Z, in which case 3 = 5 “^.. 5n"ffm'+Y • ■ • ffY+Y- Clearly 
{C,v) = {C,0){E,v). Therefore we have to solve two tasks. First, we need to find 
oi,..., Qm such that C = (7“^ ... which can be done in polynomial time [2]. 
Second, we need to find Om+i, ■ ■ ■ ,am+n such that v = Om+iei + ... + am+nen 
which is obvious. 

It follows from the discussion above that computational problems for Gf given 
by polycyclic presentation and by the deduced semidirect product are polynomial 
time equivalent. Another important property of the computed presentation is that 
the ring: 


K =q[Cu...,Cm] 

generated by matrices Ci,... is actually a field isomorphic to a subfield of F 

(because Q’s define the same action as C/i’s, but in a basis Oi,..., 0„). 


4.2. The attack. In the deduced presentation of Gp the system of conjugacy 
equations © is equivalent to the following system of equations with unknown 
C G AT* and T G Z”: 


(9) 


v{E-Bi) + tiG = t[, 
v{E — Bpf.^)+ tj^^G = 


where (C, v) represents X, {B, ti) represents bi, {B,t[) represents 6' for i = I,..., 7^2- 
To solve the system ([5]) we compute a basis Hi,... ,Hi of the field K as a vector 
space over Q. Hence, 


G — ciHi + ... + ciHi 


for some ci,..., c„ G Q and (IH) can be rewritten as: 


{ v{E — Bi) + CitiHi + ... + citiHi — ti, 

v{E — BN 2 ) E citi\[.^Hi + ... + citN 2 Hi = 

which is a system of linear equations over field Q with unknown v = (vi ,..., u„) 
and Cl,..., cj G Q. The solution of this system provides us with the key A'. 

We call this procedure as FBA2. The attack also was implemented in GAP and 
tested on the same machine. The table bellow contains results of our tests. 
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Polynomial 

KG) 

FBA2, L = 5 

FBA2, L = 100 

Time 

Success 

rate 

Time 

Success 

rate 

— X — 1 

3 

4.3 s 

100 % 

3.9 s 

100 % 

x^ - x^ -1 

7 

4.9 s 

100 % 

6.8 s 

100 % 

x"^ — x^ — \ 

10 

8.1 s 

100 % 

10.1 s 

100 % 

x^ -7x^-1 

14 

34.0 s 

100 % 

47.7 s 

100 % 

x^^ — x^ — 1 

16 

20.9 s 

100 % 

26.4 s 

100 % 

x^^ — X — 2 

22 

528.2 s 

100 % 

761.3 s 

100 % 

x^° — X — 1 

30 

164.6 s 

100 % 

208.2 s 

100 % 


5. Conclusion 

Our arguments show the following. 

• The groups of the form Up k Of can not be used as platform groups in the 
commutator key-establishment protocol. 

• It is difficult to devise a successful length-based-attack and low success rate 
does not mean much in terms of security. 

Finally we want to point out that our attack does not eliminate all polycyclic groups 
from consideration. 
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